Digital Image Archiving and Retrieval in a Mobile Device System

ABSTRACT

A computer-implemented method of managing information is disclosed. The method can include receiving a message from a mobile device configured to connect to a mobile device network (the message including a digital image taken by the mobile device and including information corresponding to words), determining the words from the digital image information using optical character recognition, indexing the digital image based on the words, and storing the digital image for later retrieval of the digital image based on one or more received search terms.

BACKGROUND

This specification discusses information organizing systems and methods,and more particularly features relating to automated archiving andretrieval of documents.

In everyday life, people frequently receive physical documents withinformation that may or may not be important, and may or may not beneeded at a later time. For example, receipts and business cards areoften received in the course of a day, and the recipient is often unsurewhether, and for how long, to save such documents. Such documents can besaved physically or scanned for storage on a computer. In either event,the saved document is typically either dropped in some location withoutany archiving meta information (e.g., dropped in a drawer or a folder),or a person must deliberately associate archiving meta information withthe document (e.g., by placing the document in a specific folderaccording to some docketing system, or by typing in information toassociate with the document saved on a computer).

SUMMARY

This specification describes methods and systems relating to documentarchiving. These methods and systems allow a user to store and readilyretrieve digital representations of physical documents. Digital imagesof physical documents can be processed using optical characterrecognition (OCR) techniques, then indexed and stored for laterretrieval. Image acquisition, OCR processing and image archiving can becombined into an end-to-end system that can facilitate management of themyriad documents encountered in everyday life (e.g., receipts, businesscards, doctor's prescriptions, tickets, contracts, etc.), and the userof this system need only take a picture to trigger the documentarchiving process in some implementations.

Users of the system can readily archive digital images of documents(with the same ease and informality of dropping a document in a drawer)and also readily retrieve the digital images using keyword searches.Digital cameras built into cell phones can be used to capture images,and OCR techniques can be used to recognize and extract relevantkeywords from these images to allow effective searches later on.Acquired document images can be delivered directly from a mobile deviceto a back-end system (e.g., mobile gateway and email server). A user ofthe system need not download images from a mobile device to a personalcomputer in order to archive and store the images, thus making imagearchiving a simple process for the user. Moreover, lower resolutionimages can also be handled using enhanced OCR techniques, includingvarious pre-processing and post-processing operations. Thus, the myriaddocuments encountered in everyday life can be readily digitized,organized, stored and retrieved quickly and efficiently.

In general, an aspect of the subject matter described in thisspecification can be embodied in a computer-implemented method thatincludes receiving a message from a mobile device configured to connectto a mobile device network, the mobile device including a digitalcamera, and the message including a digital image taken by the digitalcamera and including information corresponding to words; determining thewords from the digital image information using optical characterrecognition; indexing the digital image based on the words; and storingthe digital image for later retrieval of the digital image based on oneor more received search terms. The method can further include receivingthe one or more search terms; and retrieving the digital image based onthe one or more search terms.

The method can include validating the mobile device (e.g., based on amobile phone number and/or information associated with the receiveddigital image). Receiving the message can include receiving an emailmessage having the digital image attached; and the method can includeadding at least one of the words, and a pre-defined label correspondingto the mobile device, to the email message; and the determining,indexing and storing can be performed in an electronic mail system.

Receiving the digital image can include receiving at least two digitalimages taken of a single object in response to a single input to thedigital camera, and determining the words can include performingcorrelative optical character recognition on the at least two digitalimages to find the words. Determining the words can include performingthe optical character recognition at multiple scales.

The method can include pre-processing the digital image to improve theoptical character recognition. The pre-processing can includeidentifying a binarization threshold for the digital image by minimizingpositional variance of left and right margins of a document representedin the digital image. The pre-processing can include obtaining a graylevel at a higher resolution pixel by iteratively taking a weightedcombination of gray levels of neighboring pixels at a lower resolution.

The method can include post-processing the words to identify and correctcommon character misidentifications resulting from the optical characterrecognition. Receiving the message can include receiving an indicationof type for a document represented in the digital image, and thepost-processing can include selecting between at least two dictionarybased language models according to the indication of type for thedocument, and post-processing the words in accordance with the selecteddictionary based language model. Moreover, receiving the indication oftype can include receiving a user specified category in the message, theuser specified category selected from a group including business cardsand credit card receipts.

Other embodiments of this aspect include corresponding systems,apparatus, and one or more computer program products, i.e., one or moremodules of computer program instructions encoded on a computer-readablemedium for execution by, or to control the operation of, data processingapparatus.

An aspect of the subject matter described in this specification can beembodied in a system that includes a mobile device network; a pluralityof mobile devices configured to take digital images, connect to themobile device network, and transmit the digital images over the mobiledevice network; one or more computers configured to receive the digitalimages from the mobile devices, apply optical character recognition toextract words from the digital images, index the digital images based onthe extracted words, and store the digital images for later retrievalbased on received search terms. The one or more computers can include afirst back-end component and a second back-end component, the firstback-end component configured to receive the digital images, validatethe mobile devices and apply the optical character recognition, and thesecond back-end component configured to index the digital images andstore the digital images. The second back-end component can include anelectronic mail system.

The mobile devices can include mobile phones, and the mobile devicenetwork can include a mobile phone network. The one or more computerscan include a personal computer. The one or more computers can include asearch appliance. The one or more computers can be configured tovalidate the mobile devices based on mobile phone numbers associatedwith the mobile devices.

The one or more computers can be configured to receive the search terms,and retrieve the digital images based on the search terms. The one ormore computers can be configured to add extracted words and apre-defined label to messages including the digital images. The one ormore computers can be configured to perform correlative opticalcharacter recognition. The one or more computers can be configured toperform the optical character recognition at multiple scales.

The one or more computers can be configured to pre-process the digitalimages to improve the optical character recognition, and post-processthe extracted words to identify and correct common charactermisidentifications resulting from the optical character recognition. Theone or more computers can be configured to identify a binarizationthreshold for a digital image by minimizing positional variance of leftand right margins of a document represented in the digital image. Theone or more computers can be configured to obtain a gray level at ahigher resolution pixel by iteratively taking a weighted combination ofgray levels of neighboring pixels at a lower resolution.

The one or more computers can be configured to receive indications ofdocument type along with the digital images, select between at least twodictionary based language models according to the indications ofdocument type, and post-process the extracted words in accordance withthe selected dictionary based language model. Moreover, an indication ofdocument type can include a user specified category selected from agroup including business cards and credit card receipts.

An aspect of the subject matter described in this specification can beembodied in a system that includes a mobile device network configured totransmit digital images; a server environment configured to provideelectronic search service over a computer network; and means forconnecting the mobile device network with the server environment, themeans for connecting including means for applying optical characterrecognition to extract words from the digital images and means forproviding the extracted words and the digital images to the serverenvironment for electronic search service of the digital images via thecomputer network. The means for connecting can include means forvalidating mobile devices in the mobile device network. The means forproviding can include means for adding extracted words and a pre-definedlabel to messages including the digital images.

The means for applying can include means for performing correlativeoptical character recognition. The means for applying can include meansfor performing the optical character recognition at multiple scales. Themeans for applying can include means for pre-processing the digitalimages to improve the optical character recognition, and means forpost-processing the extracted words to identify and correct commoncharacter misidentifications resulting from the optical characterrecognition.

The means for applying can include means for identifying a binarizationthreshold for a digital image by minimizing positional variance of leftand right margins of a document represented in the digital image. Themeans for applying can include means for obtaining a gray level at ahigher resolution pixel by iteratively taking a weighted combination ofgray levels of neighboring pixels at a lower resolution. The means forapplying can include means for selecting between at least two dictionarybased language models according to received indications of documenttype, and means for post-processing the extracted words in accordancewith the selected dictionary based language model. Moreover, anindication of document type can include a user specified categoryselected from a group including business cards and credit card receipts.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the invention will be apparent from thedescription and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of an example digital image archivingsystem.

FIG. 2 is a flow chart of an example method of archiving and retrievinga digital image.

FIG. 3 is a flow chart of an example method of enhanced opticalcharacter recognition.

FIG. 4 is a schematic diagram of an example of a generic computersystem.

DETAILED DESCRIPTION

FIG. 1 is a schematic diagram of an example digital image archivingsystem 100. The system 100 includes multiple mobile devices 110 (e.g.,cell phones or personal digital assistants (PDAs)) that communicatethrough a mobile device network 120 (e.g., a private cell phone networkor wireless email network). The devices 110 are mobile in the sense thatthey can communicate using wireless transmissions (short, medium, orlong range). However, the mobile devices 110 can also include connectorsfor wired communications (e.g., a Universal Serial Bus (USB) connector).

The mobile devices 110 are configured to take digital images. Thus, amobile device 110 includes a digital camera 112. The digital camera 112can be built into a device having other functions (e.g., a mobile phoneor PDA with built in camera), or the mobile device 110 can be thedigital camera 112, which also has wireless communication capability.

The mobile device 110 can be used to take one or more digital images 132of a physical document 105. The document 105 can be any physicaldocument that includes one or more words. For example, the document 105can be a business card, an ATM (Automatic Teller Machine) receipt, acredit card purchase receipt, a doctor's prescription, a ticket fortravel (e.g., a plane ticket or railway ticket), a contract, a letter, arecipe seen in a magazine, etc. More generally, the document 105 neednot be a paper document. The document 105 can be any physical articlewith words for which one might want an archived and retrievable digitalimage, e.g., a road sign, a posted public notice, a lost pet sign, aT-shirt, etc. Note that as used herein, the term “words” includes allmanner of text information that can be identified using opticalcharacter recognition techniques, and multiple tokens can be groupedtogether and considered to be a single “word” by the system,irrespective of separating white space.

The digital image(s) 132 can be sent to a first back-end component 150in a message 130. The message 130 can be a Multimedia MessageSpecification (MMS) message including the digital image(s) 132. Othermessage formats are also possible. For example, the message 130 can bean electronic mail message.

The first back-end component 150 can connect to the mobile devicenetwork 120 through another network 140, such as the Internet.Alternatively, the first back-end component 150 can connect directly tothe mobile network 120 or be included within the mobile network 120. Forexample, the first back-end component 150 can be mobile gateway used tovalidate a cell phone 110 before the digital image(s) 132 are acceptedfor archiving.

The first back-end component 150 can include a validation engine 152configured to validate the mobile devices 110, and an OCR engine 154configured to apply optical character recognition to the digitalimage(s) 132. The first back-end component 150 can generate indexinformation 134 to add to the message 130 (e.g., by adding theinformation to a subject line of an email message), thereby associatingthe index information 134 with the digital image(s) 132.

The index information 134 includes one or more words identified in thedocument image(s) 132 using the optical character recognition. The indexinformation 134 can also include additional information, such as apre-defined label, document type information, and system stateinformation. The pre-defined label can correspond to the mobile device(e.g., the source mobile phone number), a function name associated withthe image archiving feature of the mobile device (e.g., “PIC” for“Personal Image Container” can be the label used in the mobile device'suser interface to identify the image archiving function), or both. Thedocument type information can indicate the nature of the document (e.g.,business card versus credit card receipt) and can be entered by a user(e.g., by selecting from a menu on the mobile device's user interface)or automatically determined (e.g., based on the relative vertical andhorizontal dimensions of a document represented in the digital image(s)132).

The system state information can include information such as the timeand date (e.g., time stamp) of image acquisition, transmission, receipt,or a combination of them. Further system state information can also beincluded, such as the geographic location of the mobile device at thetime of image acquisition, transmission, receipt, or a combination ofthem.

The first back-end component 150 can send the message 130, with includedindex information 134, to a second back-end component 160. The secondback-end component 160 can connect to the mobile device network 120through another network 140, such as the Internet. Alternatively, thesecond back-end component 160 can connect directly to the mobile network120 or be included within the mobile network 120.

The second back-end component 160 can include an index engine 162 and aretrieval engine 164. The index engine 162 can archive the documentimage(s) 132 based on the index information 134. The retrieval engine164 can fetch the document image(s) 132, for delivery to a networkdevice 170, based on one or more search terms received from the networkdevice 170. The network device 170 can connect to the mobile devicenetwork or the additional network 140. The network device 170 be amobile device 110 or another machine. For example, the network device170 can be a personal computer connected to the Internet and running aWeb browser.

It is to be understood that the example system 100 shown in FIG. 1 canbe implemented in multiple different ways, and the particular divisionof operational components shown is not limiting, but rather onlypresented as example. As used herein, the term “back-end component”includes both traditional back-end components (e.g., a data server) andmiddleware components (e.g., an application server). In general, thefirst and second back-end components 150 and 160 can be implementedusing one or more servers in one or more locations, i.e., a serverenvironment. For example, the first and second back-end components 150and 160 can be server machines in a publicly accessible electronic mailsystem, such as the GMAIL™ system provided by Google Inc. of MountainView, Calif.

Furthermore, it is to be understood that the message 130 can have itsformat modified between the various components of the system 100, andthus, may be considered separate messages at each stage. For example,the message received from the mobile device 110 can be in MMS format,the message received from the first back-end component 150 can be in aproprietary messaging format used between the first and secondcomponents 150 and 160, and finally the message received from the secondback-end component 160 by the network device 170 can be in HypertextMarkup Language (HTML) format.

Regardless of the formats and component configurations used, the system100 integrates the mobile devices 110, the mobile device network 120,and the back-end components 150 and 160 into one service for users ofthe mobile devices 110. Thus, for example, a user can take pictures withtheir cell phone and email the images (or send as MMS) to their emailaccount, where the images are automatically OCR'd and indexed. The usercan then access and search the images using the electronic mail system'suser interface.

FIG. 2 is a flow chart of an example method 200 of archiving andretrieving a digital image. A message is received 210 from a mobiledevice having a digital camera. The mobile device can be a cell phonefor which the user has registered the cell phone number with theiraccount in an email system, and the message can be an email sent from acell phone (e.g., to a known email address, such as archive@google.com)or an MMS sent to an email system shortcode (e.g., with a keywordindicating the archival service). The message from the mobile deviceincludes one or more digital images taken by the digital camera, and thedigital image(s) include information corresponding to words (i.e., imagedata that visually represents document text).

The mobile device can be validated 220 based on the received message.For example, a mobile gateway or the email system can validate the cellphone based on a previously employed authentication and associationmechanism. A user account can be bound to a phone number, and theauthentication and association mechanism can operate as follows. A usercan initiate a binding by filling in a form at a Web site (e.g., theemail system's Web site) specifying the user's mobile device number. Anautomated system can process the form and send an SMS (short messageservice) message to the user's mobile device for the Web request alongwith a randomly generated string. The user can then verify that stringeither on the Web or through an SMS sent back from the same mobiledevice. The user will know the string only if the mobile device belongsto the user. Alternatively, the user can initiate this binding from themobile device instead, sending a message from it to an appropriatenumber or short code with an identifier associated with the user (e.g.,as assigned by the Web site). The user's account receives a message witha string, to be verified similarly.

The words are determined 230 from the digital image information usingoptical character recognition. This can involve determining all thewords in the image or extracting only relevant keywords. For example,very common words, such as “a” and “the” can be ignored, while wordsthat occur less often in a dictionary can be ranked as more likelyrelevant. This can involve traditional techniques of simply strippingout stopwords (e.g. “and”, “for”, “a”, “the”, etc.) as used in Websearch technology. This can also involve actively identifying some wordsas likely being more relevant, such as identifying proper nouns or namedentities (e.g., “John”, “San Diego”, “Barnes & Noble”, etc.), whichlikely signify a person, place, business, etc. In some implementations,all the words can be identified, and a processing engine at the backend(e.g., the indexing engine) can handle the discrimination betweenrelevant and non-relevant words.

In some implementations, the message can include at least two images ofthe same document, and the words can be determined by performingcorrelative optical character recognition on the at least two digitalimages to find the words. For example, two digital images can be takenseparately by a user and manually grouped together for email or MMStransmission, or two digital images can be taken of a single object inresponse to a single input to the digital camera. For example, referringto FIG. 1, the digital camera 112 can have an input 114 that triggerstwo pictures to be taken in rapid succession and automatically sent tothe first back-end component 150. Note that the input 114 can also bedesigned to trigger one picture and the automatic sending.

The input 114 can be a physical button on the mobile device 110 or agraphical element in a graphical user interface of the mobile device110. The input 114 can be multifunctional, such as a side-mountedpressable thumbwheel. Alternatively, the input 114 can be dedicated tothe image archive system, such that any picture displayed on the mobiledevice's screen can be automatically transmitted for OCR-ing andarchiving in response to a single user interaction with the input 114.In any event, the input 114 can be configured to trigger sending of animage to the first back-end component 150 in response to one or two userinput actions (e.g., one or two button pushes).

Referring again to FIG. 2, the determined words can be added to thesubject line, header line or body of an email, and the full image(s) canbe stored as an attachment to the email. In addition, the email can beautomatically tagged with a pre-defined label (e.g., “PIC”). The digitalimage can be indexed 240 based on the words, and also possibly based onthe pre-defined label. Various types of word indexing can be used. Forexample, the systems and techniques described in the following patentapplications can be used: U.S. Patent Pub. No. 2005/0222985 A1, to PaulBuchheit et al., entitled “EMAIL CONVERSATION MANAGEMENT SYSTEM”, filedMar. 31, 2004 and published Oct. 6, 2005, and U.S. Patent Pub. No.2005/0223058 A1, to Paul Buchheit et al., entitled “IDENTIFYING MESSAGESRELEVANT TO A SEARCH QUERY IN A CONVERSATION-BASED EMAIL SYSTEM”, filedAug. 6, 2004 and published Oct. 6, 2005, both of which are herebyincorporated by reference. The digital image is stored 250 for laterretrieval of the digital image. Note that in some implementations, theindexing and storing operations are integrated with each other.

One or more search terms can be received 260 from a network device.These search term(s) can be entered by a user, such as in a Web browserinterface (on a mobile phone, personal computer, etc.), and sent to theimage archive system. Alternatively, these search term(s) can begenerated by a computer in response to some input. In any event, thedigital image can be retrieved 270 based on the one or more searchterms, and presented to a user or sent to another system component forfurther processing.

In some implementations, the OCR techniques handle lower resolutionimages (e.g., images from one mega pixel cameras). In addition, stepscan be taken to address issues raised by camera/lens quality, thedistance from which the document is shot, and so on. Image enhancementand super-resolution techniques can be used to pre-process the documentimage for improved OCR-ability.

FIG. 3 is a flow chart of an example method 300 of enhanced opticalcharacter recognition. A message including a digital image can bereceived 310, and the message can include an indication of type for adocument represented in the digital image. This indication of type canbe explicitly included, such as when a user notes a type for thedocument (e.g., business card versus receipt) when the picture is taken.Alternatively, the indication of type can be an aspect of the imageitself, such as the relative vertical and horizontal dimensions of adocument represented in the digital image. For example, business cardstypically have a common aspect ratio, which can be determined from adigital picture by checking for the edges of any paper document in thepicture and their relation to the text on the document. The indicationof type can also be determined by an initial OCR pass that finds somewords, and then these words can be used to indicate the document type,which can affect later OCR processing.

The digital image can be pre-processed 320 to improve optical characterrecognition. The pre-processing can involve denoising and deskewing theimage using traditional techniques. The pre-processing can involveidentifying a binarization threshold for the digital image by minimizingpositional variance of left and right margins of a document representedin the digital image. In addition, the pre-processing can employ aniterative refinement scheme that obtains the gray level at each highresolution pixel by iteratively taking a weighted combination of thegray levels of its neighboring pixels in the low resolution image.

Traditional super-resolution algorithms based on bicubic/bilinear/splineinterpolation essentially run a low pass filter on the image,eliminating sharp edges. This results in further blurring of an image,which can be undesirable when the original image was already partiallyblurred. Blurring at letter boundaries can cause degradation of OCRquality. On the other hand, edge preserving super-resolution algorithmslike nearest neighbor interpolation can cause aliasing artifacts thatconfuse the OCR engine. In contrast, the new approach described belowcan deblur, while super-sampling, without enhancing noise. Note that thewords “super-sampling” and “super-resolution” are used synonymouslyherein.

Let g(x, y)|(x, y)ε[1 . . . M, 1 . . . N], where M, N are imagedimensions, represent an observed image. Let f(x, y)|((x, y)εR²) be theunderlying true image. In this model, g is a blurred version of f, i.e.,g=f*h^(PSF), where * denotes the convolution operator, and h^(PSF)denotes the Point Spread Function (this function effectively models theblurring process). The h^(PSF) need not be known explicitly since it isknow that h^(PSF) is generally a window function performing a weightedneighborhood smoothing. As such, the Point Spread Function can bemodeled with a Gaussian function.

Considering f^((n)) as an approximation to f andg^((n))=f^((n))*h^(PSF), the equations can be rewritten as,

g=f*h ^(PSF)

G=(F·H ^(PSF))

g ^(n) =f ^(n) *h ^(PSF)

G ^((n))=(F ^((n)) ·H ^(PSF))

where upper-case letters denote Fourier Transforms. From the aboveequations,

(G−G ^((n)))=(F−F ^((n)))·H ^(PSF) or

(G−G ^((n)))·(H ^(BP))/c=(F−F ^((n)))

where c is a constant and H^(BP) is a filter. Ideally,1−(H^(BP))/c·H^(PSF)=0. However, since the Point Spread Function is alow pass filter, its Fourier Transform is usually zero at manyfrequencies, which complicates finding the function's inverse.

Hence, in practice, an iterative refinement scheme can be used:F^((n+1))=F^((n))+(G−G^((n)))·(H^(BP))/c, where H^(BP) and c are chosensuch that 1−(H^(BP))/c·H^(PSF)>0. Choosing c generally involves atradeoff. Larger c implies more noise and error tolerance, but slowerconvergence and vice versa. The initial approximation of the underlyingimage, f⁽⁰⁾, can be created via Bicubic B Spline interpolation. Thus,the iterative refinement scheme obtains the gray level at each highresolution pixel by iteratively taking a weighted combination of thegray levels of its neighboring pixels in the low resolution image.

Optical character recognition can be performed 330 on the per-processeddigital image to determine words in the digital image. The OCR operationcan be performed at multiple scales. Running the above super-resolutioncum deblurring algorithm, multiple versions of the document can becreated and OCR'd. For example, a first version at original scale, asecond version at 2× scale, and a third version at 3× scale can be fedindividually into the OCR engine and the union of the resulting wordscan be stored. The original document may have a mixture of fontsizes—the smallest font may be too small for the OCR engine torecognize. These fonts can be recognized from the higher resolution (anddeblurred) versions of the document. On the other hand, larger fontsizes in the original document, may become too large, aftersuper-resolution, for the OCR engine to recognize. These fonts can berecognized from the lower resolution versions.

In addition, irrespective of whether OCR is performed at multiplescales, often, the initial result of the optical character recognitionwill be strings of characters grouped together into words, which may ormay not be real words (e.g., the word “clip” may be read as “c1ip”, withthe lowercase letter “l” replaced by the number “1”). Thus,post-processing can be performed on the words to identify and correctcommon character misidentifications resulting from the optical characterrecognition. The post-processing can be language model based and can useone or more dictionaries.

In some implementations, multiple dictionary based language models canbe used. A selection can be made 340 between at least two dictionarybased language models according to the indication of type for thedocument. Then, the words can be post-processed 350 in accordance withthe selected dictionary based language model. In other implementations,a single dictionary based language model can be used for all images tobe OCR'd (e.g., the dictionary can be a subset of words found on theWeb).

The language based post-processing can improve the quality of OCRresults obtained from the document image. The language basedpost-processing can be understood within the context of a probabilisticframework that connects character string outputs from the OCR, withwords found in a dictionary. Note that the dictionary need not be astandard word dictionary, but can be any set of words derived from oneor more corpora.

Let w denote word (a combination of space delimited letters). Let sdenote an observed string outputted by the OCR process. Using Bayesrule,

P(w|s)=P(s|w)P(w)/P(s)

Given an observed string s, the goal is to obtain

w*=argmax_(w) P(w|s)=argmax_(w)(P(s|w)P(w))

where P(w) indicates the probability of the word w occurring, P(w|s)indicates the probability of word being actually w when it is seen byOCR as s. Thus, a w that maximizes the a posteriori probability of aword given an observed OCR output string can be sought duringpost-processing. Furthermore, the post-processing can compute w* usingtwo components: (1) A language model to estimate P(w) in the given textcontext; and (2) an OCR error model to estimate the probability ofreading word w as s, P(s|w).

The language model gives the likelihood of a word w occurring in thegiven context. For example, the occurrences of each word in a corpus oftraining documents can be counted to build a dictionary of words andword probabilities. Such a dictionary based language model can berepresented by a weighted finite state machine (WFSM) with the inputlabels as characters and accepting states corresponding to alldictionary words. Note that this example language model may not coverthe proper nouns well.

Character based language models that estimate the probability of thenext character given the string seen so far often do better with propernouns. The representation can again be a WFSM, with the following costmeasure:

C(s ₁ |c ₁ . . . c _(i-1))=−logP(s _(i) |c ₁ . . . c _(i-1))

Instead of computing the above probabilities as conditional to theentire character sequence seen so far, only a few character history needbe used. This allows coverage of a lot more words than there are in thetraining set. See, e.g., Kolak O., Resnik P., Byrne W., “A generativeprobabilistic OCR model for NLP applications”, HLT-NAACL 2003. Inaddition, n-gram word based models can be used. These models useprobability of occurrence of a word given the previous few words. Otherlanguage based models can also be used.

The error model computes the probability of the OCR engine reading aninput character sequence w as s. This too can be estimated using amachine learning approach, and an error model can be created usingtraining data, i.e., example images with input text and the OCR output.Both input and output text can be segmented into corresponding charactersegments w and s respectively. For example, this segmentation can bedone using Levenshtein edit distance. The Levenshtein distance measuresthe distance between two strings as the minimum number of operations(insertion/deletion/substitution of single character) necessary totransform one string to another. With the segmented string pairs (s,w)in hand, a weighted finite state transducer (WFST) can be computed, withinput labels corresponding to original characters and output labelsbeing the OCR output characters. See, e.g., Kolak O., Resnik P., ByrneW., “A generative probabilistic OCR model for NLP applications”,HLT-NAACL 2003. Alternatively, the edit distance approach can be usedfor computing the transition probabilities directly by measuring P(s|w)from counts above, and using the inverse as the transformation cost.

A corpus of documents with known ground truths can be used to estimatethe cost/probability of letter substitution. The actual transformations(insertion/deletion/substitution of single characters) necessary totransform each observed OCR string to the known ground truth can berecorded. The number of occurrences of each transformation is a measureof the probability/cost of that particular transformation happeningduring the OCR process. Thus, there will likely be a large number ofinstances of the letter ‘l’ being mistaken as numeral ‘1’, and henceassign a high probability to that occurrence.

Training data for computing the error model can be created byartificially generating images from text, adding noise to the generatedimages, and then generating OCR engine output from the images. Forcredit card receipts and business cards, local business listings datacan be used to learn the dictionary/language model. Additionally, usersof the system can be asked to submit document images of various types toserve as training data.

FIG. 4 is a schematic diagram of an example of a generic computer system400. The system 400 can be used for the operations described inassociation with the methods 200 and 300 according to someimplementations. For example, the system 400 may be included in any orall of the mobile devices 110, the first and second back-end components150 and 160, and the network device 170.

The system 400 includes a processor 410, a memory 420, a storage device430, and an input/output device 440. Each of the components 410, 420,430, and 440 are interconnected using a system bus 450. The processor410 is capable of processing instructions for execution within thesystem 400. In some implementations, the processor 410 is asingle-threaded processor. In other implementations, the processor 410is a multi-threaded and/or multi-core processor. The processor 410 iscapable of processing instructions stored in the memory 420 or on thestorage device 430 to display graphical information for a user interfaceon the input/output device 440.

The memory 420 stores information within the system 400. In someimplementations, the memory 420 is a computer-readable medium. In someimplementations, the memory 420 is a volatile memory unit. In someimplementations, the memory 420 is a non-volatile memory unit.

The storage device 430 is capable of providing mass storage for thesystem 400. In some implementations, the storage device 430 is acomputer-readable medium. In various different implementations, thestorage device 430 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device.

The input/output device 440 provides input/output operations for thesystem 400. In some implementations, the input/output device 440includes a keyboard and/or pointing device. In some implementations, theinput/output device 440 includes a display unit for displaying graphicaluser interfaces.

The features described can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations of them. The apparatus can be implemented in a computerprogram product tangibly embodied in an information carrier, e.g., in amachine-readable storage device or in a propagated signal, for executionby a programmable processor; and method operations can be performed by aprogrammable processor executing a program of instructions to performfunctions of the described implementations by operating on input dataand generating output. The described features can be implementedadvantageously in one or more computer programs that are executable on aprogrammable system including at least one programmable processorcoupled to receive data and instructions from, and to transmit data andinstructions to, a data storage system, at least one input device, andat least one output device. A computer program is a set of instructionsthat can be used, directly or indirectly, in a computer to perform acertain activity or bring about a certain result. A computer program canbe written in any form of programming language, including compiled orinterpreted languages, and it can be deployed in any form, including asa stand-alone program or as a module, component, subroutine, or otherunit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors ofany kind of computer. Generally, a processor will receive instructionsand data from a read-only memory or a random access memory or both. Theessential elements of a computer are a processor for executinginstructions and one or more memories for storing instructions and data.Generally, a computer will also include, or be operatively coupled tocommunicate with, one or more mass storage devices for storing datafiles; such devices include magnetic disks, such as internal hard disksand removable disks; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implementedon a computer having a display device such as a CRT (cathode ray tube)or LCD (liquid crystal display) monitor for displaying information tothe user and a keyboard and a pointing device such as a mouse or atrackball by which the user can provide input to the computer.

The features can be implemented in a computer system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system can be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include, e.g., a LAN, a WAN, and thecomputers and networks forming the Internet.

The computer system can include clients and servers. A client and serverare generally remote from each other and typically interact through anetwork, such as the described one. The relationship of client andserver arises by virtue of computer programs running on one or morecomputers and having a client-server relationship to each other.

Although a few implementations have been described in detail above,other modifications are possible. For example, any server environmentconfigured to provide electronic search service and connect to a network(i.e., any networked search engine) can be integrated with a mobiledevice network using the systems and techniques described. The serverenvironment can function as a network accessible hard drive. Moreover,the server environment need not be a traditional back-end or middlewarecomponent. The server environment can be a program installed on apersonal computer and used for electronic search of local files, or theserver environment can be a search appliance (e.g., Google™ in a Box,provided by Google Inc. of Mountain View, Calif.) installed in anenterprise network.

In addition, the logic flows depicted in the figures do not require theparticular order shown, or sequential order, to achieve desirableresults. Other operations may be provided, or operations may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherimplementations are within the scope of the following claims.

1. A computer-implemented method of managing information, the methodcomprising: receiving one or more messages from a mobile deviceconfigured to connect to a mobile device network, the mobile devicecomprising a digital camera, and the one or more messages comprising adigital image taken by the digital camera, where the digital imagerepresents a document including information corresponding to words;determining the words from the digital image information using opticalcharacter recognition, the determining comprising selecting between atleast two dictionary based language models, according to an indicationof type for the document, and post-processing the words in accordancewith the selected dictionary based language model; indexing the digitalimage based on the words; and storing the digital image for laterretrieval of the digital image based on one or more received searchterms.
 2. The method of claim 1, wherein the receiving comprisesreceiving the indication of type in the one or more messages as a userspecified category for the document.
 3. The method of claim 1, whereinthe determining comprises assessing an aspect of the digital image todetermine the indication of type for the document.
 4. The method ofclaim 1, wherein the determining comprises performing an initial opticalcharacter recognition pass to determine the indication of type for thedocument.
 5. The method of claim 1, further comprising: receiving theone or more search terms; and retrieving the digital image based on theone or more search terms.
 6. The method of claim 1, wherein receivingthe one or more messages comprises receiving an email message having thedigital image attached; the method further comprises adding at least oneof the words, and a pre-defined label corresponding to the mobiledevice, to the email message; and wherein the determining, indexing andstoring are performed in an electronic mail system.
 7. The method ofclaim 1, wherein the receiving comprises receiving at least two digitalimages taken of a single object in response to a single input to thedigital camera, and the determining comprises performing correlativeoptical character recognition on the at least two digital images to findthe words.
 8. The method of claim 1, wherein the determining comprisespre-processing the digital image to improve the optical characterrecognition.
 9. The method of claim 8, wherein the pre-processingcomprises identifying a binarization threshold for the digital image byminimizing positional variance of left and right margins of a documentrepresented in the digital image.
 10. The method of claim 8, wherein thepre-processing comprises obtaining a gray level at a higher resolutionpixel by iteratively taking a weighted combination of gray levels ofneighboring pixels at a lower resolution.
 11. A system comprising: amobile device network; a plurality of mobile devices configured to takedigital images, connect to the mobile device network, and transmit thedigital images over the mobile device network; one or more computersconfigured to receive the digital images from the mobile devices, selectbetween at least two dictionary based language models according toindications of document types for the digital images, apply opticalcharacter recognition to extract words from the digital images, indexthe digital images based on the extracted words, and store the digitalimages for later retrieval based on received search terms.
 12. Thesystem of claim 11, wherein the one or more computers are configured toreceive the indications of document types as user specified categoriesfor the digital images.
 13. The system of claim 11, wherein the one ormore computers are configured to assess an aspect of the digital imagesto determine the indications of document types.
 14. The system of claim11, wherein the one or more computers are configured to perform aninitial optical character recognition pass to determine the indicationsof document types.
 15. The system of claim 11, wherein the one or morecomputers comprise a first back-end component and a second back-endcomponent, the first back-end component configured to receive thedigital images, validate the mobile devices and apply the opticalcharacter recognition, and the second back-end component configured toindex the digital images and store the digital images.
 16. The system ofclaim 15, wherein the second back-end component comprises an electronicmail system.
 17. The system of claim 11, wherein the one or morecomputers are configured to receive the search terms, and retrieve thedigital images based on the search terms.
 18. The system of claim 11,wherein the one or more computers are configured to associate apre-defined label with the digital images.
 19. The system of claim 11,wherein the one or more computers are configured to identify abinarization threshold for a digital image by minimizing positionalvariance of left and right margins of a document represented in thedigital image.
 20. The system of claim 11, wherein the one or morecomputers are configured to obtain a gray level at a higher resolutionpixel by iteratively taking a weighted combination of gray levels ofneighboring pixels at a lower resolution.